Concerning the problem that the existing Voice Activity Detection (VAD) algorithms based on Hidden Markov Model (HMM) were poor to track noise, a method using Baum-Welch algorithm was proposed to train the noise with different characteristics, and the corresponding noise model was generated to establish a library. When voice activity was detected, depending on the measured background noise of the speech, the voice was dynamically matched to a noise model in the library. Meanwhile, in order to meet real-time requirements of speech signal processing, reduce the complexity of the speech parameter extraction, the threshold was improved to ensure the inter-frame correlation of the speech signal. Under different noise environments, the improved algorithm performance was tested and compared with Adaptive Multi-Rate (AMR), G.729B of the International Telecommunications Union (ITU-T). The test results show that the improved algorithm can effectively improve the accuracy of detection and noise tracking ability in real-time voice signal processing.